Convergence of Constant Step Stochastic Gradient Descent for Non-Smooth Non-Convex Functions

This paper studies the asymptotic behavior of constant step Stochastic Gradient Descent for minimization an unknown function, defined as expectation a non convex, smooth, locally Lipschitz random function. As gradient may not exist, it is replaced by certain operator: reasonable choice to use element Clarke subdifferential function; another output celebrated backpropagation algorithm, which popular amongst practioners, and whose properties have recently been studied Bolte Pauwels. Since chosen operator in general mean has assumed literature that oracle function available. first result, shown this such needed almost all initialization points algorithm. Next, small size regime, interpolated trajectory algorithm converges probability (in compact convergence sense) towards set solutions particular differential inclusion: subgradient flow. Finally, viewing iterates Markov chain transition kernel indexed size, invariant distribution converge weakly inclusion tends zero. These results show when small, with large probability, eventually lie neighborhood critical